-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run test suite in parallel #2507
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
paulcadman
force-pushed
the
parallel-fast-tests
branch
from
November 14, 2023 11:21
26df769
to
b96f6b7
Compare
These are used by tests and the call to getCurrentDir mean that any tests that use these functions cannot be executed in parallel
findRootAndChangeDir is used in tests, the use of getCurrentDir means that tests cannot be run in parallel.
The idea is to use this to synchronise access to filesystem resources
These functions are copied from path-0.9.5 so we will remove them when we upgrade.
paulcadman
force-pushed
the
parallel-fast-tests
branch
from
November 16, 2023 10:37
ef9dd6b
to
0bc4c2b
Compare
janmasrovira
approved these changes
Nov 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR makes the compiler pipeline thread-safe so that the test suite can be run in parallel.
This is achieved by:
{get, set, with}CurrentDir
functions.NB: Locking is disabled for the main compiler target, as it is single threaded they are not required.
Run test suite in parallel
To run the test suite in parallel you must add
--ta '+RTS -N -RTS'
to your stack test arguments. For example:The
-N
instructs the Haskell runtime to choose the number of threads to use based on how many processors there are on your machine. You can use-Nn
to see the number of threads ton
.These flags are already set in the Makefile when you or CI uses
stack test
.Locking
The Haskell package filelock is used for locking. File locks are used instead of MVars because Juvix code does not control when new threads are created, they are created by the test suite. This means that MVars created by Juvix code will have no effect, because they are created independently on each test-suite thread. Additionally the resources we're locking live on the filesystem and so can be conveniently tagged by path.
FileLock
The filelock library is wrapped in a FileLock effect:
juvix/src/Juvix/Data/Effect/FileLock/Base.hs
Lines 6 to 8 in e6dca22
There is an IO interpreter that uses filelock and an no-op interpreter that just runs actions unconditionally.
TaggedLock
To make the file locks simpler to use a TaggedLock effect is introduced:
juvix/src/Juvix/Data/Effect/TaggedLock/Base.hs
Lines 5 to 11 in e6dca22
And convenience function:
juvix/src/Juvix/Data/Effect/TaggedLock.hs
Line 28 in e6dca22
This allows an action to be locked, tagged by a directory that may or may not exist. For example in the following code, an action is performed on a directory
root
that may delete the directory before repopulating the files. So the lockfile cannot be stored in theroot
itself.juvix/src/Juvix/Extra/Files.hs
Lines 55 to 60 in e6dca22
Pipeline
As noted above, we only use locking in the test suite. The main app target pipeline is single threaded and so locking is unnecessary. So the interpretation of locks is parameterised so that locking can be disabled
juvix/src/Juvix/Compiler/Pipeline/Run.hs
Line 64 in e6dca22